Methods and systems for creation of hanging protocols using graffiti-enabled devices

ABSTRACT

Certain embodiments of the present invention provide methods and systems for hanging protocol generation using gesture recognition. Certain embodiments provide a method for creating a hanging protocol based on gesture input in a clinical environment. The method includes specifying a hanging protocol specification using gesture-based input. The method also includes translating the gesture-based input into a hanging protocol. The method further includes facilitating display of clinical information based on the hanging protocol. Certain embodiments provide a gesture detection system. The system includes a sensor surface configured to detect gesture-based input made on the sensor surface. The gesture-based input specifies a hanging protocol layout. The system also includes a processor configured to identify the gesture-based input and translate the gesture to a corresponding hanging protocol definition for display of image and clinical data.

RELATED APPLICATIONS

[Not Applicable]

FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[Not Applicable]

MICROFICHE/COPYRIGHT REFERENCE

[Not Applicable]

BACKGROUND OF THE INVENTION

The present invention generally relates to improving healthcareapplication workflow. In particular, the present invention relates touse of gesture recognition to improve healthcare application workflow.

A clinical or healthcare environment is a crowded, demanding environmentthat would benefit from organization and improved ease of use of imagingsystems, data storage systems, and other equipment used in thehealthcare environment. A healthcare environment, such as a hospital orclinic, encompasses a large array of professionals, patients, andequipment. Personnel in a healthcare facility must manage a plurality ofpatients, systems, and tasks to provide quality service to patients.Healthcare personnel may encounter many difficulties or obstacles intheir workflow.

In a healthcare or clinical environment, such as a hospital, a largenumber of employees and patients may result in confusion or delay whentrying to reach other medical personnel for examination, treatment,consultation, or referral, for example. A delay in contacting othermedical personnel may result in further injury or death to a patient.Additionally, a variety of distraction in a clinical environment mayfrequently interrupt medical personnel or interfere with their jobperformance. Furthermore, workspaces, such as a radiology workspace, maybecome cluttered with a variety of monitors, data input devices, datastorage devices, and communication device, for example. Clutteredworkspaces may result in efficient workflow and service to clients,which may impact a patient's health and safety or result in liabilityfor a healthcare facility.

Data entry and access is also complicated in a typical healthcarefacility. Speech transcription or dictation is typically accomplished bytyping on a keyboard, dialing a transcription service, using amicrophone, using a Dictaphone, or using digital speech recognitionsoftware at a personal computer. Such dictation methods involve ahealthcare practitioner sitting in front of a computer or using atelephone, which may be impractical during operational situations.Similarly, for access to electronic mail or voice messages, apractitioner must typically use a computer or telephone in the facility.Access outside of the facility or away from a computer or telephone islimited.

Thus, management of multiple and disparate devices, positioned within analready crowded environment, that are used to perform daily tasks isdifficult for medical or healthcare personnel. Additionally, a lack ofinteroperability between the devices increases delay and inconvenienceassociated with the use of multiple devices in a healthcare workflow.The use of multiple devices may also involve managing multiple logonswithin the same environment. A system and method for improving ease ofuse and interoperability between multiple devices in a healthcareenvironment would be highly desirable.

In a healthcare environment involving extensive interaction with aplurality of devices, such as keyboards, computer mousing devices,imaging probes, and surgical equipment, repetitive motion disordersoften occur. A system and method that eliminates some of the repetitivemotion in order to minimize repetitive motion injuries would be highlydesirable.

Healthcare environments, such as hospitals or clinics, include clinicalinformation systems, such as hospital information systems (HIS) andradiology information systems (RIS), and storage systems, such aspicture archiving and communication systems (PACS). Information storedmay include patient medical histories, imaging data, test results,diagnosis information, management information, and/or schedulinginformation, for example. The information may be centrally stored ordivided at a plurality of locations. Healthcare practitioners may desireto access patient information or other information at various points ina healthcare workflow. For example, during surgery, medical personnelmay access patient information, such as images of a patient's anatomy,that are stored in a medical information system. Alternatively, medicalpersonnel may enter new information, such as history, diagnostic, ortreatment information, into a medical information system during anongoing medical procedure.

In current information systems, such as PACS, information is entered orretrieved using a local computer terminal with a keyboard and/or mouse.During a medical procedure or at other times in a medical workflow,physical use of a keyboard, mouse or similar device may be impractical(e.g., in a different room) and/or unsanitary (i.e., a violation of theintegrity of an individual's sterile field). Re-sterilizing after usinga local computer terminal is often impractical for medical personnel inan operating room, for example, and may discourage medical personnelfrom accessing medical information systems. Thus, a system and methodproviding access to a medical information system without physicalcontact would be highly desirable to improve workflow and maintain asterile field.

Imaging systems are complicated to configure and to operate. Often,healthcare personnel may be trying to obtain an image of a patient,reference or update patient records or diagnosis, and orderingadditional tests or consultation. Thus, there is a need for a system andmethod that facilitate operation and interoperability of an imagingsystem and related devices by an operator.

In many situations, an operator of an imaging system may experiencedifficulty when scanning a patient or other object using an imagingsystem console. For example, using an imaging system, such as anultrasound imaging system, for upper and lower extremity exams,compression exams, carotid exams, neo-natal head exams, and portableexams may be difficult with a typical system control console. Anoperator may not be able to physically reach both the console and alocation to be scanned. Additionally, an operator may not be able toadjust a patient being scanned and operate the system at the consolesimultaneously. An operator may be unable to reach a telephone or acomputer terminal to access information or order tests or consultation.Providing an additional operator or assistant to assist with examinationmay increase cost of the examination and may produce errors or unusabledata due to miscommunication between the operator and the assistant.Thus, a method and system that facilitates operation of an imagingsystem and related services by an individual operator would be highlydesirable.

Additionally, image volume for acquisition and radiologist reviewcontinues to increase. PACS imaging tools have increased in complexityas well. Thus, interactions with standard input devices (e.g., mouse,trackball, etc.) have become increasingly more difficult. Radiologistshave complained about a lack of ergonomics with respect to standardinput devices, such as a mouse, trackball, etc. Scrolling through largedatasets by manually cine-ing or scrolling, repeated mouse movements,and other current techniques have resulted in carpel tunnel syndrome andother repetitive stress syndromes. Radiologists have not been able toleverage other, more ergonomic input devices (e.g., joysticks, videoeditors, game pads, etc.), because the devices are not customconfigurable for PACS and other healthcare application interactions.

Tablets, such as Wacom tablets, have been used in graphic arts but haveno current applicability or interactivity with other applications, suchas healthcare applications. Handheld devices, such as personal digitalassistants or pocket PCs, have been used for general scheduling andnote-taking but have not been adapted to healthcare use or interactionwith healthcare application workflow.

Devices facilitating gesture-based interaction typically affordmotion-based interactions whereby a user writes or motions a characteror series of characters that corresponds to a specific softwarefunction. Gesture recognition algorithms typically attempt to recognizea pattern or character gestured by the user. Typical gesture recognitionsystems focus on recognition of the gestured character alone. In thecase of an image magnify, a user must gesture, for example, the letter“z.” The gesture-enabled image processing or display system responds bygenerically zooming the image. Unfortunately, the system is unaware of aspecific level of zoom that the user is requesting from this gesturebased interaction. If a user would like to further zoom in, he/she mustrepeatedly gesture the letter “z” to zoom to the appropriate level. Suchrepetition may not only be time consuming, but may also be a physicaldrain on the user.

As discussed above, clinicians, especially surgeons, are challenged withmaintaining a sterile environment when using conventional computerdevices such as a mouse and keyboard. Several approaches have beenproposed to address the desire to maintain a sterile clinicalenvironment, such as use of a sterile mouse/keyboard, gesturerecognition, gaze detection, a thin-air display, voice command, etc.However, problems remain with these approaches. Voice command andcontrol appears to be a viable solution but, due to proximity issues andpresence of multiple people in an operating room providing confusion andinterference, use of voice command and control may not be very practicalor effective. Use of a thin-air display still suffers from very complexinteraction with computer(s) in the clinical environment.

Radiologists traditionally want less and more intuitive interaction withcomputers for using PACS applications. In most cases, interactionproblems are compounded by poor graphical user interfaces for functionssuch as zooming, cine, window scroll (which may involve a morecontinuous interaction), etc. In most cases, radiologists use a regularmouse or a scroll mouse and experimentally attempt to vary thespeed/velocity of scroll/cine, etc.

A graffiti character set may be used with a user interface to allow aradiologist to directly interact with PACS by drawing/writing graffiticharacters/gestures on an image and thereby provide a user interfacewithout a separate graphical user interface. However, for zooming,scrolling or cine, users will have to write the corresponding charactersmultiple times, adding complexity to the process.

A hanging protocol is a set of display rules for presenting, formattingand otherwise organizing images on a display device of a PACSworkstation, for example. A display rule is a convention for presentingone or more images in a particular temporal and/or spatial layout orsequence. For example, a hanging protocol may include a set ofcomputer-readable instructions (or display rules, for example) thatdirect a computer to display a plurality of images in certain locationson a display device and/or display the plurality of images in a certainsequence or order. In another example, a hanging protocol may include aset of computer-readable instructions that direct a computer to place aplurality of images in multiple screens and/or viewports on a displaydevice. In general, a hanging protocol may be employed to present aplurality of images for a diagnostic examination of a patient anatomyfeatured in the images.

A hanging protocol may direct, for example, a PACS workstation todisplay an anterior-posterior (“AP”) image adjacent to a lateral imageof the same anatomy. In another example, a hanging protocol may directPACS workstation to display the AP image before displaying the lateralimage. In general, a hanging protocol dictates the spatial and/ortemporal presentation of a plurality of images at PACS workstation.

A hanging protocol differs from a default display protocol (“DDP”). Ingeneral, a DDP is a default workflow that applies a series of imageprocessing functions to image data. The image processing functions areapplied to the image data in order to present an image (based on theimage data) to a user. The image processing functions alter theappearance of image data. For example, an image processing function mayalter the contrast level of an image.

DDPs typically include processing steps or functions that are appliedbefore any diagnostic examination of the images. For example, processingfunctions may be applied to image data in order to enhance featureswithin an image (based on the image data). Such processing functions caninclude any software-based application that may alter a visualappearance or representation of image data. For example, a processingfunction can include any one or more of flipping an image, zooming in animage, panning across an image, altering a window and/or level settingin a representation of the image data, and altering a contrast and/orbrightness setting in a representation of the image data.

DDPs are usually based on a type of imaging modality used to obtain theimage data. For example, image data obtained with a C-arm imaging devicein general or a particular C-arm imaging device may have a same orsimilar DDP applied to the image data. In general, a DDP attempts topresent image data in a manner most useful to many users. Conversely,applying a hanging protocol to image data does not alter the appearanceof an image (based on the image data), but instead dictates how theimage(s) is(are) presented, as described above.

Hanging protocols are currently created using one or more of a mouse,keyboard and graphical user interface (“GUI”) components. Suchcomponents are used to specify an image box type (for example, stack,sheet, volume, etc.). The components are used to specify parameters forimage sets to be loaded in the image box. The components are also usedto set image view presets and/or presentation state, for example.Current hanging protocol creation involves multiple input devices, and auser must switch between keyboard and mouse to create a hangingprotocol.

Thus, there is a need for systems and methods to improve healthcareworkflow using gesture recognition and other interaction. Furthermore,systems and methods for more streamlined gesture-based control would behighly desirable. Systems and methods for improved creation andmanagement of hanging protocols would also be highly desirable.

BRIEF SUMMARY OF THE INVENTION

Certain embodiments of the present invention provide methods and systemsfor hanging protocol generation using gesture recognition.

Certain embodiments provide a method for creating a hanging protocolbased on gesture input in a clinical environment. The method includesspecifying a hanging protocol specification using gesture-based input.The method also includes translating the gesture-based input into ahanging protocol. The method further includes facilitating display ofclinical information based on the hanging protocol.

Certain embodiments provide a computer-readable medium having a set ofinstructions for execution on a computer. The computer-readable mediumincludes a sensor routine for detecting gesture-based input for ahanging protocol specification. The computer-readable medium alsoincludes a translation routine for translating the gesture-based inputto a corresponding hanging protocol definition.

Certain embodiments provide a gesture detection system. The systemincludes a sensor surface configured to detect gesture-based input madeon the sensor surface. The gesture-based input specifies a hangingprotocol layout. The system also includes a processor configured toidentify the gesture-based input and translate the gesture to acorresponding hanging protocol definition for display of image andclinical data.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 illustrates an information input and control system forhealthcare applications and workflow used in accordance with anembodiment of the present invention.

FIG. 2 shows an example of an interface and graffiti used in accordancewith an embodiment of the present invention.

FIG. 3 illustrates a flow diagram for a method for gesture-basedinteraction with a healthcare application in accordance with anembodiment of the present invention.

FIGS. 4A-4B depict examples demonstrating how a size and/or a positionof a gesture can affect a size of a corresponding action according toembodiments of the present invention.

FIG. 5 illustrates a flow diagram for a method for associating a gesturewith a healthcare application function in accordance with an embodimentof the present invention.

FIG. 6 illustrates a pressure-sensitive gesture-based interaction systemin accordance with an embodiment of the present invention.

FIG. 7 illustrates a flow diagram for a method for associating apressure with a gesture to execute a healthcare application function inaccordance with an embodiment of the present invention.

FIG. 8 illustrates creation of a hanging protocol using agraffiti-enabled workstation in accordance with an embodiment of thepresent invention.

FIG. 9 illustrates translation of graffiti input to a hanging protocolin accordance with an embodiment of the present invention.

FIG. 10 illustrates a flow diagram for a method for graffiti-basedhanging protocol creation in accordance with an embodiment of thepresent invention.

The foregoing summary, as well as the following detailed description ofcertain embodiments of the present invention, will be better understoodwhen read in conjunction with the appended drawings. For the purpose ofillustrating the invention, certain embodiments are shown in thedrawings. It should be understood, however, that the present inventionis not limited to the arrangements and instrumentality shown in theattached drawings.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 illustrates an information input and control system 100 forhealthcare applications and workflow used in accordance with anembodiment of the present invention. The system 100 includes aninterface 110, a communication link 120, and a healthcare application130. The components of the system 100 may be implemented in software,hardware, and/or firmware, for example. The components of the system 100may be implemented separately and/or integrated in various forms.

The communication link 120 serves to connect the interface 110 and thehealthcare application 130. The link 120 may a cable or other wire-basedlink, a data bus, a wireless link, an infrared link, and/or other dataconnection, for example. For example, the communication link 120 may bea USB cable or other cable connection. Alternatively or in addition, thecommunication link 120 may include a Bluetooth, WiFi, 802.11, or otherwireless communication device, for example. The communication link 120and interface 110 allow a user to input and retrieve information fromthe healthcare application 130 and to execute functions at thehealthcare application 130 and/or other remote system.

The interface 110 is a user interface, such as a graphical userinterface, that allows a user to input information, retrieveinformation, activate application functionality, and/or otherwiseinteraction with the healthcare application 130. As illustrated in FIG.2, the interface 110 may be a tablet-based interface with a touchscreencapable of accepting stylus, pen, keyboard, and/or human touch input,for example. For example, the interface 110 may be used to drivehealthcare applications and may serve as an interaction device and/or asa display to view and interact with screen elements, such as patientimages or information. The interface 110 may execute on and/or beintegrated with a computing device, such as a tablet-based computer, apersonal digital assistant, a pocket PC, a laptop, a notebook computer,a desktop computer, a cellular phone, and/or other handheld orstationary computing system. The interface 110 facilitates wired and/orwireless communication and provides audio, video and or other graphicaloutput, for example.

The interface 110 and communication link 120 may include multiple levelsof data transfer protocols and data transfer functionality. Theinterface 110 and communication link 120 may support a plurality ofsystem-level profiles for data transfer, such as an audio/video remotecontrol profile, a cordless telephony profile, an intercom profile, anaudio/video distribution profile, a headset profile, a hands-freeprofile, a file transfer protocol, a file transfer profile, and/or animaging profile. The communication link 120 and the interface 110 may beused to support data transmission in a personal area network (PAN) orother network.

In an embodiment, graffiti-based stylus or pen interactions, such asgraffiti 240 shown in FIG. 2, may be used to control functionality atthe interface 110 and/or healthcare application 130 via the interface110 and communication link 120. Graffiti and/or other strokes may beused to represent and/or trigger one or more commands, commandsequences, workflow, and/or other functionality at the interface 110and/or healthcare application 130, for example. That is, a certainmovement or pattern of a cursor displayed on the interface 110corresponds to or triggers a command or series of commands at theinterface 110 and/or healthcare application 130, for example.Interactions triggered by graffiti and/or other gesture or stroke may becustomized for healthcare application(s) and/or for particular user(s)or group(s) of user(s), for example. Graffiti/stroke(s) may beimplemented in a variety of languages instead of or in addition toEnglish, for example. Graffiti interactions or shortcuts may be mappedto keyboard shortcuts, program macros, and/or specific interactions, forexample.

The healthcare application 130 may be a healthcare software application,such as an image/data viewing application, an image/data analysisapplication, an annotation and/or reporting application, and/or otherpatient and/or practice management application. The healthcareapplication 130 may include hardware, such as a Picture Archiving andCommunication System (PACS) workstation, advantage workstation (AW),PACS server, image viewer, personal computer, workstation, server,patient monitoring system, imaging system, or other data storage orprocessing device, for example. The interface 110 may be used tomanipulate functionality at the healthcare application 130 including butnot limited to image zoom (e.g., single or multiple zoom), applicationand/or image reset, display window/level setting, cine/motion, magicglass (e.g., zoom eyeglass), image/document annotation, image/documentrotation (e.g., rotate left, right, up, down, etc.), image/documentflipping (e.g., flip left, right, up, down, etc.), undo, redo, save,close, open, print, pause, indicate significance, etc. Images and/orinformation displayed at the healthcare application 130 may be affectedvia the interface 110 via a variety of operations, such as pan, cineforward, cine backward, pause, print, window/level, etc.

In an embodiment graffiti or other gesture or indication may becustomizable and configurable by a user and/or administrator, forexample. A user may create one or more strokes and/or functionalitycorresponding to one or more strokes, for example. In an embodiment, thesystem 100 may provide a default configuration of strokes andcorresponding functionality. A user, such as an authorized user, maycreate his or her own graffiti and/or functionality, and/or may modifydefault configuration of functionality and corresponding graffiti, forexample. A user may combine a sequence or workflow ofactions/functionality into a single gesture/graffiti, for example.

In an embodiment, a password or other authentication, such as voice orother biometric authentication, may also be used to establish aconnection between the interface 110 and the healthcare application 130via the communication link 120. Once a connection has been establishedbetween the interface 110 and the healthcare application 130, commandsmay be passed between interface 110 and the healthcare application 130via the communication link 120.

In operation, for example, a radiologist, surgeon or other healthcarepractitioner may use the interface 110 in an operating room. The surgeonmay request patient data, enter information about the current procedure,enter computer commands, and receive patient data using the interface110. To request patient data or enter computer commands, the surgeon“draws” or otherwise indicates a stroke or graffiti motion on theinterface 110. The request or command is transmitted from the interface110 to the healthcare application 130 via the communication link 120.The healthcare application 130 then executes command(s) received fromthe interface 110. If the surgeon requests patient information, thehealthcare application 130 retrieves the information. The healthcareapplication 130 may then transmit the patient information to theinterface 110 via the communication device 120. Alternatively or inaddition, the information may be displayed at the healthcare application130. Thus, requested information and/or function result may be displayedat the interface 110, healthcare application 130, and/or other display,for example.

In an embodiment, when a surgeon or other healthcare practitionersterilizes before a procedure, the interface 110 may be sterilized aswell. Thus, a surgeon may use the interface 110 in a more hygienicenvironment to access information or enter new information during aprocedure, rather than touch an unsterile keyboard or mouse for thehealthcare application 130.

In certain embodiments, a user may interact with a variety of electronicdevices and/or applications using the interface 110. A user maymanipulate functionality and/or data at one or more applications and/orsystems via the interface 110 and communication link 120. The user mayalso retrieve data, including image(s) and related data, from one ormore system(s) and/or application(s) using the interface 110 andcommunication link 120.

For example, a radiologist carries a wireless-enabled tablet PC. Theradiologist enters a radiology reading room to review or enter imagedata. A computer in the room running a healthcare application 130recognizes the tablet PC interface 110 via the communication link 120.That is, data is exchanged between the tablet PC interface 110 and thecomputer via a wireless communication link 120 to allow the interface110 and the healthcare application 130 to synchronize. The radiologistis then able to access the healthcare application 130 via the tablet PCinterface 110 using strokes/gestures at the interface 110. Theradiologist may view, modify, and print images and reports, for example,using graffiti via the communication link 120 and tablet PC interface110. The interface 110 enables the radiologist to eliminate excessclutter in a radiology workspace by replacing use of a telephone,keyboard, mouse, etc. with the interface 110. The interface 110 andcommunication link 120 may simplify interaction with a plurality ofapplications/devices and simplify a radiologist's workflow through useof a single interface point and simplified gestures/strokes representingone or more commands/functions.

In certain embodiments, interface strokes may be used to navigatethrough clinical applications such as a picture archiving andcommunication system (PACS), a radiology information system (RIS), ahospital information system (HIS), and an electronic medical record(EMR). A user's gestures/graffiti may be used to execute commands in asystem, transmit data to be recorded at the system, and/or retrievedata, such as patient reports or images, from the system.

In certain embodiments, the system 100 may include voice command andcontrol capability. For example, spoken words may be converted to textfor storage and/or display at a healthcare application 130.Additionally, text at the healthcare application 130 may be converted toaudio for playback to a user at the interface 110 via the communicationlink 120. Dictation may be facilitated using voice recognition softwareon the interface 110 and/or the healthcare application 130. Translationsoftware may allow dictation as well as playback of reports, lab data,examination notes, and image notes, for example. Audio data may bereviewed in real-time in stereo sound via the system 100. For example, adigital sound file of a patient heartbeat may be reviewed by a physicianremotely through the system 100.

The communication link 120 and interface 110 may also be used tocommunicate with other medical personnel. Certain embodiments mayimprove reporting by healthcare practitioners and allow immediateupdating and revising of reports using gestures and/or voice commands.Clinicians may order follow-up studies at a patient's bedside or duringrounds without having to locate a mouse or keyboard. Additionally,reports may be signed electronically, eliminating delay or inconvenienceassociated with a written signature.

FIG. 3 illustrates a flow diagram for a method 300 for gesture-basedinteraction with a healthcare application in accordance with anembodiment of the present invention. First, at step 310, one or moregestures are mapped to one or more functionality. For example, a gestureindicating a rudimentary representation of an anatomy, such as a breast,may retrieve and display a series of breast exam images for a patient.Other exemplary gestures and corresponding functionality may include,but are not limited to, a diagonal line from left to right to zoom in onan image, a diagonal line from right to left to zoom out on an image, acounterclockwise semi-circle to rotate and 3D reformat an imagecounterclockwise, a clockwise semi-circle to rotate and 3D reformat animage clockwise, a series of circles may indicate a virtual colonoscopysequence, and/or a gesture indicating a letter “B” may correspond toautomatic bone segmentation in one or more images.

In certain embodiments, a series or workflow of functionality may becombined into a signal stroke or gesture. For example, a stroke madeover an exam image may automatically retrieve related historical imagesand/or data for that anatomy and/or patient. A stroke made with respectto an exam may automatically cine through images in the exam andgenerate a report based on those images and analysis, for example. Astroke may be used to provide structured and/or standard annotation inan image and/or generate a report, such as a structured report, forimage analysis. Strokes may be defined to correspond to standard codes,such as Current Procedural Terminology (CPT), InternationalClassification of Diseases (ICD), American College of Radiology (ACR),Digital Imaging and Communications in Medicine (DICOM), Health LevelSeven (HL7), and/or American National Standards Institute (ANSI) codes,and/or orders, for example. Strokes may be defined to correspond to anyfunctionality and/or series of functionality in a healthcareapplication, for example.

In an embodiment, a default configuration of strokes and functionalitymay be provided. In an embodiment, the default configuration may bemodified and/or customized for a particular user and/or group of users,for example. In an embodiment, additional stroke(s) and/or functionalitymay be defined by and/or for a user and/or group of users, for example.

At step 320, a connection is initiated between an interface, such asinterface 110, and a remote system, such as healthcare application 130.Data packets are transmitted between a remote system and an interface toestablish a communication link between the remote system and theinterface. The communication link may also be authenticated using voiceidentification or a password, for example. The connection may beestablished using a wired or wireless communication link, such ascommunication link 120. After the communication link has beenestablished, a user may interact with and/or affect the remote systemvia the interface.

Next, at step 330, a user gestures at the interface. For example, theuser enters graffiti or other stroke using a pen, stylus, finger,touchpad, etc., at an interface screen. In an embodiment, a mousingdevice may be used to gesture on an interface display, for example. Thegesture corresponds to a desired action at the remote system. Thegesture may also correspond to a desired action at the interface, forexample. A gesture may correspond to one or more commands/actions forexecution at the remote system and/or interface, for example.

Then, at step 340, a command and/or data corresponding to the gesture istransmitted from the interface to the remote system. If the gesture wererelated to functionality at the interface, then the gesture is simplytranslated into a command and/or data at the interface. In certainembodiments, a table or other data structure stores a correlationbetween a gesture and one or more commands, actions, and/or data whichare to be input and/or implemented as a result of the gesture. When agesture is recognized by the interface, the gesture is translated to thecorresponding command and/or data for execution by a processor and/orapplication at the interface and/or remote system.

At step 350, the command and/or data is executed and/or entered at theremote system. In an embodiment, if a command and/or data were intendedfor local execution at the interface, then the command and/or data isexecuted and/or entered at the interface. Data may be entered,retrieved, and/or modified at the interface, such as the interface 110,and/or the remote system, such as the healthcare application 130, basedon the gesture, for example. An application and/or functionality may beexecuted at the remote system and/or interface in response to thegesture, for example. In an embodiment, a plurality of data and/orfunctionality may be executed at the remote system and/or interface inresponse to a gesture, for example.

Next, at step 360, a response is displayed. A response may be displayedat the interface and/or at the remote system, for example. For example,data and/or application results may be displayed at the interface and/orremote system as a result of command(s) and/or data executed and/orentered in response to a gesture. A series of images may be shown and/ormodified, for example. Data may be entered into an image annotationand/or report, for example. One or more images may be acquired,reviewed, and/or analyzed according to one or more gestures, forexample. For example, a user using a pen to draw a letter “M” or othersymbol on an interface display may result in magnification of patientinformation and/or images on an interface and/or remote system display.

In certain embodiments, graffiti/gesture based interactions can be usedas symbols for complex, multi-step macros in addition to 1-to-1 keyboardor command mappings. A user may be afforded greater specificity bymodifying a graffiti/gesture-based command/action based on a size andposition of character/gesture performed. For example, a level of zoomthat a user desires with respect an image can be determined by the sizeof the character “z” he/she gestures on the image. If he/she is lookingto zoom in to a medium degree, he/she gestures a medium sized “z”, andso forth. The position of the gesture may also modify a gesture. Forexample, zooming in on a lower left quadrant of an image window mayallow the user to affect and zoom in on the lower quadrant of the image,and so forth.

FIG. 4A depicts examples demonstrating how a size of a gesture canaffect a size of a corresponding action. As shown in the first panel ofFIG. 4A, the smaller “z” gesture 410 results in a smaller zoom effect415. A medium-sized “z” gesture 420 results in a medium-sized zoomeffect 425. A larger “z” gesture 430 in the third panel produces aproportionally larger zoom factor 435.

FIG. 4B depicts examples demonstrating how a position of a gesture canaffect a relative position of an image with regard to a certain gestureinteraction. As shown in FIG. 4B, a small zoom or “z” gesture 440 in thelower left quadrant of an image results in a small zoom of the lowerleft quadrant of the image 445. In the second panel of FIG. 4B, a smallzoom gesture 450 in the upper right quadrant of the image results in asmall zoom of the upper right quadrant of the image 455.

FIG. 5 illustrates a flow diagram for a method 500 for associating agesture with a healthcare application function in accordance with anembodiment of the present invention. At step 510, a gesture is mapped toa healthcare application function. For example, the gesture or character“z” is mapped to a zoom or magnify command in an image processing orreview application.

At step 520, the gesture-to-function mapping is modified based on anadditional characteristic associated with the gesture/graffiti. Forexample, a size of a gestured “z” is mapped to a certain degree of zoom(e.g., a “normal”-sized “z” corresponds to a certain degree of zoomwhile a smaller “z” and a larger gestured “z” correspond to an order ofmagnitude smaller and larger zoom of an image, respectively). As anotherexample, a position of a gestured “z” is mapped to a certain area ofzoom (e.g., a gestured “z” in a lower left quadrant of an imagecorresponds to a zoom of the lower left quadrant of the image and agestured “z” in an upper left quadrant of an image corresponds to a zoomof the upper left quadrant of the image). In certain embodiments, aplurality of characteristics (e.g., size and position) may be combinedto modify a gesture-to-function mapping. Additionally, although a “z”gesture and an image zoom command have been used above, it is understoodthat use of “z” and zoom is for purposes of illustration only and manyother gesture-based commands (e.g., “c” to cine a series of images, “m”to magnify an image, “s” for segmentation, “b” for bone segmentation,“w” to adjust window level, “r” to reset, drag and drop gestures, etc.)may be implemented according to embodiments of the present invention.

At step 530, the modified gesture-to-function mapping is stored forfuture use. In certain embodiments, mappings may be later modified by auser and/or tailored for a particular user and/or group of usersaccording to a profile and/or single-session modification. In certainembodiments, mappings may be dynamically created for a single-sessionuse and/or dynamically created and saved for further future use, forexample.

Certain embodiments enhance a graffiti- or gesture-based clinicalsystem, such as a PACS system, using pressure a user applies on agraffiti pen or other gesturing instrument and/or a display or othersensor to adjust a characteristic or parameter of the gesture-basedcommand, such as a velocity or repetition of a zoom, cine or scrollcommand. As an example, a user may want to cine through a stack ofimages. The user begins by writing or gesturing a character (e.g., theletter “c”) to start a manual cine. If the user wants to scroll throughthe image faster, the user applies more pressure to the gesturinginstrument, such as a graffiti pen or stylus. In certain embodiments, ifthe user applies less pressure to the instrument, scrolling slows down.The action stops when the user applies no pressure. The same processapplies to any continuous input need for scrolling or zooming or otheroperations, for example.

FIG. 6 illustrates a pressure-sensitive gesture-based interaction system600 in accordance with an embodiment of the present invention. FIG. 6shows a clinician zooming on the image with graffiti with pressuresensor. As shown in FIG. 6, a clinician 610 gestures to form a graffiticharacter 640 on a display 620 using an instrument 630. For example, theclinician 610 gestures to form a “z” on the display 620 using a stylus.The display 620 includes one or more sensors, such as a touch sensoroverlaying and/or integrated with the display surface, to detectgestures made on the display 620. The sensor(s) and display 620 transmitdetected gestures, such as a gestured “z”, to a processing unit 650. Theprocessing unit 650 may be integrated with the display 620, integratedwith a clinical information system, such as a PACS, RIS, HIS, etc.,and/or implemented separately in hardware, firmware and/or software, forexample.

The processing unit 650 receives the gesture information and translatesthe gesture to healthcare application functionality. For example, theprocessing unit 650 receives information representing a gestured “z”, asshown in FIG. 6, and maps the “z” gesture to a zoom command. Theprocessing unit 650 may also detect a degree of pressure applied by theuser 610 to the instrument 630 and/or to the display 620. The degree ofpressure may be used to modify the gesture-to-command mapping, forexample. For example, a degree of pressure on the stylus corresponds toa degree of zoom applied to the displayed image (e.g., for each degreeof increased pressure, zooming in on the image is increased). Theprocessing unit 650 then transmits the zoom command to a healthcareapplication, such as a PACS image review application.

FIG. 7 illustrates a flow diagram for a method 700 for associating apressure with a gesture to execute a healthcare application function inaccordance with an embodiment of the present invention. At step 710, agesture made using a gesture instrument is mapped to a healthcareapplication function. For example, the gesture or character “z” madeusing a pen, stylus or other detectable instrument is mapped to a zoomor magnify command in an image processing or review application.

At step 720, the gesture-to-function mapping is modified based onpressure applied to the instrument and/or to the display by the userwhen making the gesture/graffiti. For example, a relative amount ofpressure (e.g., compared to a “normal” or no excess amount of pressure)applied to the instrument and/or to the display when making the gestured“z” is mapped to a certain degree of zoom (e.g., a normal or normalizeddegree of pressure corresponds to a certain degree of zoom while asmaller degree of pressure and a larger degree of pressure made whengesturing “z” correspond to an order of magnitude smaller and largerzoom of an image, respectively). In certain embodiments, a plurality ofcharacteristics may be combined to modify a gesture-to-function mapping.Additionally, although a “z” gesture and an image zoom command have beenused above, it is understood that use of “z” and zoom is for purposes ofillustration only and many other gesture-based commands (e.g., “c” tocine a series of images, “m” to magnify an image, “s” for segmentation,“b” for bone segmentation, “w” to adjust window level, “r” to reset,drag and drop gestures, etc.) may be implemented according toembodiments of the present invention.

At step 730, the modified gesture-to-function mapping is executed and aresult displayed to the user. In certain embodiments, mappings may belater modified by a user and/or tailored for a particular user and/orgroup of users according to a profile and/or single-sessionmodification. In certain embodiments, mappings may be dynamicallycreated for a single-session use and/or dynamically created and savedfor further future use, for example.

Certain embodiments use graffiti-enabled display devices to createand/or manage hanging protocols on a PACS client workstation and/orother terminal where a user can use graffiti handwriting technologyalong with a display device to draw a hanging protocol layout andwrite/select image box type and parameters, for example.

FIG. 8 illustrates creation of a hanging protocol using agraffiti-enabled workstation in accordance with an embodiment of thepresent invention. A user draws a desired layout on an interface 800using a graffiti pen and/or other gesture-based input device. As shownin FIG. 8, rectangular shapes identify image boxes 810, 820, 830. Atiled image box 830 represents a sheet with an up-count equal to anumber of tiles. The tiles can be equally spaced and have equal size,for example. The system 800 allows the user to write a specification inan image box 810, 820, 830, where the system 800 recognizes thespecification and converts the specification into an image boxdefinition for the hanging protocol.

For example, a specification may include an image box type (e.g., stack,sheet, volume, etc.). A specification may include an upcount (e.g., 1,2, 4, 6, 8, etc.). A specification may include a modality (e.g.,magnetic resonance, computed tomography, X-ray, ultrasound, etc.).Additionally, a specification may include a body part (e.g., head, neck,chest, limb, etc.). A specification may further include a list of one ormore procedures involved. A specification may also include a comparisonindication (e.g., yes or no). In certain embodiments, a specificationincludes one or more series selection parameters (for example, one ormore parameter names and values, such as parameter=slice thickness,value=3 mm, etc.). A specification may include other configurationinformation and/or settings, for example.

A graffiti- or gesture-enabled hanging protocol tool translates thecreated specification into a hanging protocol definition, as illustratedin FIG. 2. As described above, gesture-based/graffiti input is mapped toone or more commands, parameters and/or other input for a clinicalsystem. The input is used to generate the hanging protocol for displayof images at a clinical system, such as a PACS workstation. Image boxspecifications 810, 820, 830 are translated to display image boxes 910,920, 930. The specification from FIG. 8 determines how data is displayedin the image boxes 910, 920, 930 of FIG. 9.

FIG. 10 illustrates a flow diagram for a method 1000 for graffiti-basedhanging protocol creation in accordance with an embodiment of thepresent invention. At step 1010, a user generates gesture or graffitiinput using an input, such as a hand, a graffiti pen and/or otherinstrument. The input may be generated on a touch screen,graffiti-enabled display, thin-air display, etc. The gesture/graffitiinput describes a hanging protocol specification, for example.

At step 1020, the gesture/graffiti input is translated into a hangingprotocol. For example, the specification reflected in thegesture/graffiti input is translated into a hanging protocol fordisplaying clinical images and other data at a workstation. At step1030, clinical images and other data are displayed at a clinicalworkstation, such as a PACS or RIS/PACS workstation, in accordance withthe defined hanging protocol.

Thus, certain embodiments provide an improved or simplified workflow fora clinical environment, such as radiology or surgery. Certainembodiments allow a user to operate a single interface device to accessfunctionality and transfer data via gestures and/or other strokes.Certain embodiments provide a system and method for a user toconsolidate the workflow of a plurality of applications and/or systemsinto a single interface.

Certain embodiments of the present invention provide increased efficientand throughput for medical personnel, such as radiologists andphysicians. Systems and methods reduce desktop and operating roomclutter, for example, and provide simplified interaction withapplications and data. Repetitive motion injuries may also be reduced oreliminated.

Thus, certain embodiments leverage portable input devices, such astablet and handheld computing devices, as well as graffiti/gesture-basedinteractions with both portable and desktop computing devices, tointeract with and control healthcare applications and workflow. Certainembodiments provide an interface with graffiti/gesture-based interactionallowing users to design custom shortcuts for functionality andcombinations/sequences of functionality to improve healthcare workflowand simplify user interaction with healthcare applications.

Certain embodiments facilitate interaction through a stylus- and/ortouch-based interface with graffiti/gesture-based interaction that allowusers to easily design custom shortcuts for existing menu items and/orother functionality. Certain embodiments facilitate definition and useof gestures in one or more languages. Certain embodiments provideergonomic and intuitive gesture shortcuts to help reduce carpel tunnelsyndrome and other repetitive injuries. Certain embodiments provide useof a portable interface to retrieve, review and diagnose images at theinterface or another display. Certain embodiments allow graffiti orother gesture to be performed directly on top of an image or document tomanipulate the image or document.

Certain embodiments reduce repetitive motions and gestures to affordmore precise interactions. Certain embodiments allow a user to add morespecific control to gestural input through additional cues based on sizeand position of the gesture-based input.

Certain embodiments provide a sterile user interface for use by surgeonsand other clinicians operating in a sterile environment. Certainembodiments provide a gesture-based system that can be used inconjunction with a regular monitor and/or thin-air display to displayand modify image and/or other clinical data. Certain embodiments providean intuitive user interface without reliance on a graphical userinterface. Pressure on a pen or other similar instrument can be variedto change a characteristic of a clinician application function, such asa velocity of scroll, zoom, cine, etc. Certain embodiments combine PACS,pressure sensitive instrumentation and graffiti to provide clinicians aneffective user interface.

Certain embodiments provide free-form hanging protocol creation bydrawing a layout and writing associated parameters. Certain embodimentsprovide a simple user interface facilitating reduced searching forcontrols on the screen. Certain embodiments allow creation of equal sizeor flexible size layouts, for example. Certain embodiments simulate theuse of pen and paper to provide a more natural way for a user to createhanging protocol layout and specification.

While the invention has been described with reference to certainembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted withoutdeparting from the scope of the invention. In addition, manymodifications may be made to adapt a particular situation or material tothe teachings of the invention without departing from its scope.Therefore, it is intended that the invention not be limited to theparticular embodiment disclosed, but that the invention will include allembodiments falling within the scope of the appended claims.

1. A method for creating a hanging protocol based on gesture input in aclinical environment, said method comprising: specifying a hangingprotocol specification using gesture-based input, wherein said hangingprotocol specification is defined by drawing a desired layout of one ormore image boxes on an interface using said gesture-based input tocreate a new hanging protocol by providing the desired layout of one ormore image boxes and a gesture written specification for each image boxon the interface for translation into a hanging protocol for imagedisplay; automatically translating said gesture-based input into ahanging protocol, wherein said translating said gesture-based input intoa hanging protocol includes converting said hanging protocolspecification into one or more image box definitions for said hangingprotocol; and facilitating display of clinical information based on saidhanging protocol.
 2. The method of claim 1, wherein said hangingprotocol specification comprises an image box type and at least oneparameter.
 3. The method of claim 1, wherein said gesture-based inputincludes a gesture component and at least one of a size component and aposition component modifying said gesture component.
 4. The method ofclaim 1, wherein said gesture-based input corresponds to a sequence ofhealthcare application functions for execution at a remote system. 5.The method of claim 1, wherein said gesture-based input includes apressure component comprising at least one of a pressure applied to aninstrument used to make said gesture and a pressure applied to a sensorsurface.
 6. The method of claim 1, further comprising customizing atranslation between said gesture-based input describing said hangingprotocol specific and said hanging protocol for at least one of a userand a group of users.
 7. A computer-readable medium having a set ofinstructions for execution on a computer, said set of instructionscomprising: a sensor routine for detecting gesture-based input for ahanging protocol specification, wherein said hanging protocolspecification is defined by drawing a desired layout of one or moreimage boxes on an interface using said gesture-based input; atranslation routine for automatically translating said gesture-basedinput to a corresponding hanging protocol definition, wherein saidtranslating said gesture-based input into a hanging protocol definitionincludes converting a hanging protocol specification defined by saidgesture-based input into one or more image box definitions for saidhanging protocol to create a new hanging protocol by providing thedesired layout of one or more image boxes and a gesture writtenspecification for each image box on the interface for translation into ahanging protocol for image display; and a display routine facilitatingdisplay of clinical information based on said hanging protocoldefinition.
 8. The computer-readable medium of claim 7 wherein saidhanging protocol specification comprises an image box type, an upcount,a modality, a body part, a list of one or more procedures, a comparisonindication and one or more series selection parameters.
 9. Thecomputer-readable medium of claim 7 wherein said gesture-based inputfurther includes a characteristic associated with said gesture-basedinput.
 10. The computer-readable medium of claim 9 wherein saidtranslation routine modifies said hanging protocol definitioncorresponding to said gesture-based input using said characteristicassociated with said gesture-based input.
 11. The computer-readablemedium of claim 9 wherein said characteristic includes at least one of aposition and a size of said gesture-based input.
 12. A gesture detectionsystem, said system comprising: a sensor surface configured to detectgesture-based input made on said sensor surface, said gesture-basedinput specifying a hanging protocol layout, wherein said hangingprotocol layout is defined by drawing a desired layout of one or moreimage boxes on an interface using said gesture-based input to create anew hanging protocol by providing the desired layout of one or moreimage boxes and a gesture written specification for each image box onthe interface for translation into a hanging protocol for image display;a processor configured to identify said gesture-based input andautomatically translate said gesture to a corresponding hanging protocoldefinition for display of image and clinical data, wherein saidtranslating said gesture-based input into a hanging protocol definitionincludes converting a hanging protocol specification defined by saidgesture-based input into one or more image box definitions for saidhanging protocol, said processor to facilitate display of clinicalinformation based on said hanging protocol.
 13. The system of claim 12wherein said gesture-based input specifying said hanging protocol layoutcomprises an image box type, an upcount, a modality, a body part, a listof one or more procedures, a comparison indication and one or moreseries selection parameters.
 14. The system of claim 12 furthercomprising a graffiti input instrument for generating said gesture-basedinput on said sensor surface.
 15. The system of claim 14 wherein atleast one of a pressure exerted on said sensor surface and a pressureexerted on said instrument modifies said gesture-based input.
 16. Thesystem of claim 12 wherein said gesture-based input further includes acharacteristic associated with said gesture-based input, saidcharacteristic modifying said hanging protocol definition correspondingto said gesture-based input.
 17. The system of claim 12 furthercomprising a display for displaying at least one of images and clinicaldata according to said hanging protocol definition.